Incremental Feature Selection and l1 Regularization for Relaxed Maximum-Entropy Modeling

We present an approach to bounded constraintrelaxation for entropy maximization that corresponds to using a double-exponential prior or `1 regularizer in likelihood maximization for log-linear models. We show that a combined incremental feature selection and regularization method can be established for maximum entropy modeling by a natural incorporation of the regularizer into gradientbased feature selection, following Perkins et al. (2003). This provides an efficient alternative to standard `1 regularization on the full feature set, and a mathematical justification for thresholding techniques used in likelihood-based feature selection. Also, we motivate an extension to n-best feature selection for linguistic features sets with moderate redundancy, and present experimental results showing its advantage over `0, 1-best `1, `2 regularization and over standard incremental feature selection for the task of maximum-entropy parsing.1